Method and apparatus for enhancing stereo vision

ABSTRACT

A method and apparatus for segmenting an image are provided. The method may include the steps of clustering pixels from one of a plurality of images into one or more segments, determining one or more unstable segments changing by more than a predetermined threshold from a prior of the plurality of images, determining one or more segments transitioning from an unstable to a stable segment, determining depth for one or more of the one or more segments that have changed by more than the predetermined threshold, determining depth for one or more of the one or more transitioning segments, and combining the determined depth for the one or more unstable segments and the one or more transitioning segments with a predetermined depth of all segments changing less than the predetermined threshold from the prior of the plurality of images.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/993,414, filed May 30, 2018 to El Dokor et al., currently pending,which is a continuation of U.S. patent application Ser. No. 15/136,904to El Dokor et al., filed Apr. 23, 2016 to El Dokor et al., now U.S.Pat. No. 10,037,602, which is a continuation of U.S. patent applicationSer. No. 14/226,858 to El Dokor et al., filed Mar. 27, 2014, titledMethod and Apparatus for Enhancing Stereo Vision Through ImageSegmentation, now U.S. Pat. No. 9,324,154, which is a continuation ofU.S. patent application Ser. No. 13/316,606 to El Dokor et al. filedDec. 12, 2011 titled Method and Apparatus for Enhanced Stereo Vision,now U.S. Pat. No. 8,718,387, which is a continuation of U.S. patentapplication Ser. No. 13/297,029 filed 15 Nov. 2011 to Cluster et at.titled Method and Apparatus for Fast Computational Stereo, now U.S. Pat.No. 8,705,877, which is in turn a continuation of U.S. patentapplication Ser. No. 13/294,481 filed 11 Nov. 2011 to El Dokor et al.titled Method and Apparatus for Enhanced Stereo Vision, now U.S. Pat.No. 9,672,609. The '606 application is also a continuation of U.S.patent application Ser. No. 13/297,144 filed 15 Nov. 2011 to Cluster etal. titled Method and Apparatus for Fast Computational Stereo, now U.S.Pat. No. 8,761,509, which is in turn a continuation of U.S. patentapplication Ser. No. 13/294,481 filed 11 Nov. 2011 to El Dokor et al.titled Method and Apparatus for Enhanced Stereo Vision, now U.S. Pat.No. 9,672,609.

BACKGROUND OF THE INVENTION

Systems and methods for generating depth maps for images have sufferedfrom lack of precision and requirements for great computing resources.Additionally, specialized hardware is often required in order togenerate such a depth map. Imprecision in generation of such a depth mapmay result in poor resolution of acquired images and difficulty inidentifying precise locations of objects within those depth maps.Without such precise identification of these locations, later processingof these images and objects may result in a reduced ability to rely onthese locations and objects for additional processing.

Therefore, it would be desirable to present a method and apparatus thatovercomes the drawbacks of the prior art.

SUMMARY OF THE INVENTION

In accordance with various embodiments of the present invention, amethod and apparatus is provided for stabilizing segmentation and depthcalculations. The inventors of the present invention have presented, inU.S. patent application Ser. No. 13/025,038, titled “Method andApparatus for Performing Segmentation of an Image”, filed Feb. 10, 2011to El Dokor et al., Ser. No. 13/025,055, titled “Method and Apparatusfor Disparity Computation in Stereo Images: filed February 10 to ElDokor et al., and Ser. No. 13/025,070, titled “Method and Apparatus forDetermining Disparity of Texture”, filed Feb. 10, 2011 to El Dokor etal., the entire contents of each of these applications beingincorporated herein by reference, a case for describing various types ofsegments, labeled as stable or unstable segments, used for developing adisparity map. This is described as being accomplished by matching suchsegments with their appropriate counterparts between the two images in astereo image sequence. Building on the implementation described in theabove-mentioned applications, in accordance with various embodiments ofthe present invention, a series of criteria is presented for updatingthe various segments, specifically with the goal of efficient andaccurate depth map updating.

As is described in the '038, '055 and '070 applications, it ismeaningful to look only at one or more changes associated with a givenstereo image sequence to produce a subsequent depth map and not theentire image. Thus, rather than recomputing an entirely new depth mapfor each pair of stereo images over time, only changes betweenconsecutive frames are computed and integrated into one composite depthmap. This process is not only computationally more efficient thanrecomputing the complete depth map for each stereo frame pair, it isalso more accurate for matching, since only regions with significantchanges are being matched in any given frame or sequence of frames. Thisis an altogether novel approach to computational stereo as previousattempts have been faced with a significant amount of computationalcomplexity, problems with limiting a candidate space of depthcalculations, and a nebulous set of features at best to extract from,without these features being robust to significant changes in thescene's quality or even overall color scheme.

In accordance with various embodiments of the present invention, aframework with which such an approach can be accomplished is provided,defining various types of regions and segments that are associated withsuch an approach. Also presented are other relevant aspects and featuresto develop a set of factors that can improve the accuracy ofsegmentation and the accuracy of the depth map itself, by presenting ashallow-depth of field concept with two different realizations.

Still other objects and advantages of the invention will in part beobvious and will in part be apparent from the specification anddrawings.

The invention accordingly comprises the several steps and the relationof one or more of such steps with respect to each of the other steps,and the apparatus embodying features of construction, combinations ofelements and arrangement of parts that are adapted to affect such steps,all as exemplified in the following detailed disclosure, and the scopeof the invention will be indicated in the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the invention, reference is made tothe following description and accompanying drawings, in which:

FIG. 1 is a flowchart diagram depicting relationships between a numberof segment classifications in accordance with an embodiment of theinvention;

FIG. 2 is a flowchart diagram depicting processing for updating a depthmap in accordance with a depth map in accordance with the invention;

FIG. 3 is a state diagram depicting a relationship for movement betweendifferent pixel segments in accordance with an embodiment of theinvention;

FIG. 4 is a flowchart diagram depicting 3D segmentation of a depth map;

FIG. 5 is a flowchart diagram depicting 3D segmentation of a depth mapusing gray world and color world segments;

FIG. 6 depicts a number of steps for segmentation and depth calculationin accordance with an embodiment of the invention; and

FIG. 7 depicts the relationship of a MEM-mounted lens and an imagesensor in accordance with an additional embodiment of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

One or more embodiments of the invention will now be described, makingreference to the following drawings in which like reference numbersindicate like structure between the drawings.

An image scene may be organized into a series of homogenized segments,or “regions”, defined by their color, texture properties or otherrelevant properties. Each of these regions defines one or more segmentsin the image. The challenge for creating a disparity map (and hence, adepth map) from such segments lies in matching these segments againsttheir correct counterparts in a secondary image (the other image in, forexample, a stereo pair of images). To accomplish that, an even biggerchallenge is associated with segmenting these images into segments ofmeaningful context. The above-mentioned '038, '055 and '070 applicationspresent a more detailed discussion of the different clusteringtechniques that may be associated with defining segments, and associatedmatching techniques. One of the most important assumptions that was madein these applications, and that carries through to the presentinvention, is that changes in the image, in general, are very gradual.While there are regions of abrupt changes, they are few in relationshipto an entire image sequence. Most regions exhibit this well-behaved andgradual change. Such gradual transition allows exploitation ofredundancy of data between frames to:

1) Stabilize segments, since most segments mostly do not change, i.e. nodifferences in their pixel values between consecutive frames, based onintra-frame metrics; and

2) Stabilize a subsequent depth map, in that if segments associated witha given disparity value have not changed, then the subsequent depth mapalso should not change. This makes perceptual sense since changes in thescene's depth should be reflected by changes in the observed field ofview.

Given these two general observations, in accordance with embodiments ofthe present invention, a more coherent approach to disparity-based depthcompute is provided, in which the depth map is iteratively improvedthrough segmentation, followed by depth computation. Temporal andspatial stability criteria of various segments become crucial todetermining disparity updates, and hence depth map updates, since absentof such criteria a temporal-based approach can not be implemented, andthe scene's temporal redundancy can not be successfully exploited.Tracking across such a broad range of features ensures that changes inthe image are correctly accounted for and integrated into the depth map.

This novel approach to segmentation/depth map calculation allows ahighly efficient and accurate depth map to be produced and enables areal-time implementation that can be embedded on smaller platforms,including (system-on-a-chip) SoC platforms, where an existing embeddedGPGPU can provide the parallel computation that is preferred toimplement this approach.

Given the importance of stability criteria, embodiments of the presentinvention define various segment types that enhance scene understandingthrough such criteria, and exploit such criteria to understand scenechanges across frames and within the same frame. Thus, as is shown inFIG. 1, five different segment types may be defined. These segments arepreferably classified in two groups, non-residual segments 110 andresidual segments 120. Residual segments 120 further may comprise stablesegments 112, partially stable segments 114 and mesostable segments 116.Similarly, residual segments may comprise unstable segments 122 andunsegmented background 124. As is further shown in FIG. 1, pixels maymove between these segments as will be described in accordance with theflowchart set forth in FIG. 2. Therefore, at step 210, a previouscluster map is copied and stored in memory as a current cluster map.Next, at step 215, all pixels that have changed in color by more thanTps (or frames) are declared residual. Then, at step 220, one or moreresidual pixels may be clustered together to form one or more unstablesegments (as such because they are just formed). At step 225, depth iscalculated for all partially stable and unstable segments. Then, thedepth computed for the unstable segments is considered at step 230, andif the calculated depth is determined to be valid, the correspondingunstable segment is moved to the mesostable range. Mesostable segmentsremaining after Tms frames are moved to the stable range at step 235,and at step 240 mesostable segmentation is refined so that segments withthe best color match between stable and mesostable ranges are merged.Finally at step 245, non-residual segments are reevaluated forstability.

Table 1, below, depicts the various segment types, and a description ofcharacteristics of that segment type. As can be seen, stable segmentshave little or no change, while partially stable segments have a smallamount of change, but the segment is still considered generally stable.Mesostable segments are transitioning from an unstable classification toa more stable classification, while an unstable segment has enoughmotion that it may have to be resegmented, and may be a result of any ofthe other classifications. Other/background segments include all otherpixels that cannot be placed into a segment with other pixels.

TABLE I Description of various segments and pixels. Pixel SegmentSegment Classification Classification Subcategory DescriptionNon-residual Mostly Stable Stable Segments Segments that have had littlePixels Segments or no change Partially Stable Segments with a smallSegments amount of change Mesostable Segments that have partiallySegments stabilized from being unstable Residual Unstable Segments withsignificant Pixels Segments enough motion that have to be resegmentedOther/Background Every other type of pixel, including background,filtered out pixels, noise, etc.

As is noted, pixels therefore may move between classifications basedupon change associated therewith. A state diagram depicting movementpaths between various classifications is shown at FIG. 3. As can be seenin FIG. 3 (as well as with the arrow indicators in FIG. 1), a stablesegment 310 can remain a stable segment, can be changed to a partiallystable segment 320 if a small amount of change is determined, or can bedeclared an unstable segment 340 if significant change is determined.Similarly, a partially stable segment 320 can become a stable segment310 if it remains partially stable for a predetermined period of time,can become a mesostable segment 330 if some change below a predeterminedthreshold is detected, and can become an unstable segment 340 ifsignificant change greater than a predetermined threshold is determined.Similarly, a mesostable segment 330 can become a partially stablesegment 320 or a stable segment 310 if no change is determined, and itcan become an unstable segment 340 if a lot of change is determined.Finally, various pixels in the unstable segments may be declared “other”(or unsegmented) 350 if they do not appear to be able to be segmented.The predetermined threshold may be predefined, or learned in accordancewith processing of data.

Therefore, pixels may be classified into one of the two general classes:residual pixels, i.e. pixels that have changed in the image asdetermined in accordance with one or more intra-frame metrics, andnon-residual pixels representing pixels that have not changed, alsobased on such metrics. Segments undertake the overall process describedearlier: they may be first created, by identifying pixels that areresidual. They then may migrate to states of mesostability or stability,depending on the associated criteria. A depth may be computed andassociated with such segments, and then a second depth-basedsegmentation step may be implemented. By default, any pixel or group ofpixels that have not been assigned to stable or mesostable, are assignedto unstable.

Organizing a scene into various segments is preferably predicated uponthe concept that neighboring pixels generally exhibit similar behavior,and hence generally do belong to the same segment. This behavior mayinvolve characteristics of motion, color changes, texture features, orany combination of features. The exception to this notion lies at objectboundaries and/or at depth discontinuities.

Once objects, or segments, are identified as stable or unstable, thenatural progression is towards cluster numbers that stabilize theprocess over time, so that only changes in images are accounted for.This general theoretical approach, though very different in its details,is widely exploited in video encoding (Wiegand, Sullivan, Bjøntegaard, &Luthra, 2003) at a much more basic level, in which segments areconsidered for dominant features for texture or motion-based coding. Themost substantial contribution and difference here is the explicitdefinition of different segment types, their lifecycle, and theassociated pixel states, aspects of the work that are not present invideo coding. Additionally video coding techniques do not attempt toglean or extract depth or even associated segments with various depths.The invention as set forth in one or more embodiments of the presentinvention also exploits advances in GPU computing to parallelize theprocess of clustering and scene organization.

The utilization of image segments for calculating depth and iterativelyimproving segmentation through gleaning scene queues of perceptualrelevance allows disparity computation to take place in a very efficientmanner. A feature, such as motion, that can be a very dominant featurein scene analysis, can be extracted from mesostable segments, i.e.,segments transitioning between an unstable state and a stable one. Localchanges in the image that are associated with motion may be clusteredand tracked through residual segmentation first. Disparities may then becomputed by only matching such segments with ones that represent similarmesostable changes and ignoring all other pixels. Hence, the searchspace that is associated with disparity matching is greatly reduced, andmatching accuracy is enhanced. Once depth is computed, a new depth mapcan be reclustered based on combining stable segments with recentlydepth-computed mesostable segments.

FIG. 4 depicts a process associated with this inventive approach. 2DSegmentation may be first attempted at step 410 on the image sequenceand regions are broken into homogeneous segments ofcolor/texture/orientation and scale. For orientation and scale, it ispreferred that a variant of the complex wavelet transform be used toextract texture information. A depth map is preferably next computed instep 420, based on the segmentation step. In real-time, unstable pixelsmay be clustered together to form residual regions that are segmentedseparately in step 415 and then have their depth computed next at step425. The newly computed regions' residual depth map may then be combinedwith the stable depth map at step 430. The overall composite map maythen be reclustered by combining mesostable segments with stablesegments in a step that also involves segmentation based on depth.

Similar to the process noted above, one or more color spaces may becombined together to produce meaningful segmentation processing. Inaccordance with another embodiment of the present invention, not onlyare residual segments computed, but a scene may be broken down into twoor more orthogonal scenes: one of high Chroma (color world) and one oflow Chroma (gray world). The two scenes may then be segmented, and thenthe steps set forth in FIG. 4 may also be implemented for eachsegmentation. The result is a more complete and comprehensive depth map.In the gray world, intensity becomes a dominant feature, with extractedscale-space and frequency features being used in disparitydecomposition. As a result, the task of depth map computation may bebeen divided into two tasks, depending on the individual pixel's (andassociated regions') dominant features: for color pixels, hue is a goodrepresentation of color. For low-Chroma pixels, intensity helps indifferentiating the pixel. Gradients and structures that are associatedwith these features can be extracted, as well as the scales that areassociated with such gradients. However, the fundamental approachdescribed earlier remains unchanged, namely: performing segmentationand/or residual segmentation, and then computing depth on both, andcombining the results in an overall composite depth map.

Once the gray world depth map has been created, it can be easilycombined and fused with the high-Chroma depth map, presented earlier.FIG. 5 represents the algorithm with the two processes runningsimultaneously, 510 representing the processing of FIG. 4 applied to thecolor world data and 520 representing the processing of FIG. 4 appliedto the gray world. In the particular embodiment of FIG. 5, data need notbe shared between the two processes, but rather, the final result ispreferably combined to produce a composite depth map 530.

FIG. 6 depicts a three-frame sequence in which depth computation may beperformed, in two aspects: for stable segments as well as the unstablesegments. After initial segmentation of unstable segments, a depthcomputation is performed on partially stable segments as well asmesostable segments. Therefore, as is shown in FIG. 6, row 610 depicts aseries of source images at frames f_(i), f_(i+1), f_(i+2). In thisparticular example, an idealized movement of a user's arm is shownbetween frames in a sequence, these frames not necessarily beingconsecutive. Employing the residual 2D segmentation process describedabove, row 620 depicts such residual segmentation, and in particulardepicts the positioning of the hand in the prior frame and the newposition overlaid. Areas of the new position that overlap the oldposition are further segmented. Row 630 then shows the results of depthcomputation, figuring the depth of each of the segments shown in theresidual segmentation process of row 620. Row 640 depicts performance of3D segmentation as described, and finally depth composition is shown atrow 650. Thus, in accordance with FIG. 6, a three-frame sequence inwhich depth computation is performed is shown. The depth computation maybe performed in two aspects: for stable segments as well as for unstablesegments. After initial segmentation of unstable segments, depthcomputation may be performed on partially stable segments as well asmesostable segments.

In accordance with another embodiment of the present invention, basedupon the determination that video sequences are well behaved, then onemay make the additional useful assumption that any associatedsegmentation map and any additional subsequently computed maps arelikely also well behaved. Thus, even when confronted with a givenpartially stable segment whose disparity is to be recalculated, awell-behaved segment allows the assumption that a newly computeddisparity for that segment is likely in the neighborhood of the old onefrom the previous frame, as the segment may be tracked across one ormore frames. As such, it is possible to define two second level types ofstability for a particular partially stable segment:

-   -   1. Major stability, indicating that very few pixels have        changed. Thus, it may be determined that there has not been        enough change to warrant a reevaluation of the segment's        disparity, i.e. new depth calculation.    -   2. Minor stability, indicating that enough of the pixels have        changed that depth is to be recalculated.        If segment stability does not fall into either of the above        mentioned categories, and it is therefore determined that the        segment is unstable, then pixels associated with this segment        are preferably classified as unstable and the entire        segmentation process may be repeated.

All pixels in corresponding images are preferably marked with theirrespective states. This is particularly important since matchingrelevant pixels with each other across frames requires a mechanism withwhich such pixels are correctly marked. From an implementationperspective, marking pixels during disparity decomposition in a manneras described in the above-mentioned '038, '055 and '070 applications,while matching, is an effective interpretation of this approach. Markedout pixels cannot contribute to further matching during the disparitydecomposition step, and so false positives are reduced. Disparitydecomposition, as described in the above-mentioned '038, '055 and '070applications can be conducted left-to-right or right-to-left, and pixelswith existing and accurate disparity can be marked out to reduce thesearch space that is associated with the disparity decomposition.

Block-Based GPU Clustering and Implementation on a Discrete GPU or anIntegrated GPU of a System on a Chip

GPU technology allows for launching of multiple simultaneously processedthreads for processing video images. The threads are preferably managedby a thread scheduler, each thread adapted to work on one or more pixelsin an image. See (NVIDIA: CUDA compute unified device architecture,prop. guide, version 1.1, 2007) for more details. Groups of threads maybe combined to process pixel blocks with having rectangular or otherdesirable dimensions. One or more methods for clustering of such pixelsemploying GPU-based implementations are described in the above-mentioned'038, '055 and '070 applications, in which block based statistics arefirst computed and then combined across blocks. As a direct result ofthis process, localized statistics representing intermediate results ofvarious clusters at GPU block-level (from the GPU architecture) areavailable. Additionally, one or more global statistics constitutinglocalized combinations of all the localized block-level statistics arealso available. This means that for any given cluster, both localized aswell as global statistical information is available for segmentation.This same paradigm would also apply to GPUs that are integrated onboardan SoC, like ARM's MALI or Imgtec's SGX PowerVR or any other GPU or GPUIP representation involving the utilization of SIMD architectures andcalling functions.

When performing segmentation of an image, one of the biggest challengesinvolves finding the correct optimizations of local and global metricsthat are associated with a given segment or cluster to allow forappropriate clustering of different segments in an image. For any givenresidual segment, clustering an existing stable segment not onlyrequires global statistics, but also local ones. This is especially truefor larger segments, in which global statistics may vary drasticallyfrom local ones, especially in the presence of a color or texturegradient. Two segments may have very different overall globalstatistics, but they may also have local statistics that are well suitedto allow them to cluster together. Utilizing the GPU's intrinsicproperties involving launching blocks of threads to operate oncontiguous data, adjacent blocks that belong to two different clustersmay be very similar and can be used to combine clusters together. Thiscan also apply for tracking changes in blocks of data that areassociated with larger segments. Utilizing block-based statistics allowssegments to remain relatively stable as they transition between states,and as they temporally progress and evolve through an image sequence.

The thread scheduler can also be modified through configurationsettings, to account for such a computational stereo approach.

The inventive approach specifically utilizes the GPU's thread scheduleras a means of imposing local metrics on image segmentation. As a result,local metrics become an intrinsic consequence of the GPU architecture,provided appropriate implementation in either software or hardware orboth.

A GPU-based architecture can then be designed to optimize theutilization of the GPU's thread scheduler for segmentation. ArithmeticLogic Units (ALUs) can be used to process adjacent pixels in an image,local changes being associated with thread blocks and global changesbeing represented as combinations of such local changes. Merging at theblock level before merging on the grid level, i.e. entire image, allowsall threads in a block to write to fewer locations, mitigating manyatomic operations. Atomic operations are a common bottleneck associatedwith computer vision algorithms being implemented on GPGPUarchitectures.

Shallow Depth of Field

Depth of field is that part of the field of view of a camera thatcontains the sharpest edges (the amount of sharpness in the scene), see(Peterson, 2010). Peterson defines three major factors contributing todepth of field:

The focal length of the lens

The distance between the sensor and the object in the field-of-view

Aperture of the lens

A shallow depth of field has the effect of blurring objects outsideregions with high sharpness (i.e. outside regions in focus). Theblurring effect can aid in identifying background objects. Featuresassociated with scale and frequency can be exploited to mitigate thebackground objects, reduce scene clutter, and improve depth computationaccuracy.

Various embodiments of the present invention include at least twoapproaches to mitigate excessive FLOPs computation based on exploitingproperties of the field-of-view through blurring the background with ashallow depth of field. In doing so, the background selectively standsin contrast to the foreground, and can be removed through theutilization of large-scale low pass filtering kernels or selectivewavelet-based filtering, since background blurriness becomes a salientfeature of the scene and can be exploited. During residual segmentation,having a shallow depth of field enhances matching foreground-segmentedobjects, since erroneous background objects are minimized with a moreblurred background model. There are many techniques to highlight thefundamental differences between the foreground and background in a scenewith a shallow depth of field. Techniques like PCA, SVM, or training aNeural Network can be used to detect such regions' features. There alsoexists prior work in the literature on sharpness metrics that can alsobe applied in this case to enhance foreground-backgrounddiscriminability. The two methods for reducing such depth of field willnow be described.

Space-Frequency Feature Extraction for Segment Matching

One inventive approach for matching segments or image regions is toutilize space-frequency features utilizing tools such as waveletdecomposition. Therefore, in accordance with an embodiment of thepresent invention, the following process may be employed. First, acandidate segment is preferably defined, {tilde over (s)}_(R)(x,y),whose disparity is being evaluated. An operator F{ψ_(R)(x,y)} is alsodefined such that ψ_(R)(x,y) is a basis function. A space-frequencydecomposition may therefore be defined as:R _({tilde over (S)}) _(R) (x,y)={tilde over (S)} _(R)(x,y)*F{ψ_(R)(x,y)}

As noted above, such features allow a background model to be extractedand utilized in matching and segmentation. With a background that isrelatively uniform and smooth, frequency-space decomposition can then beapplied to the scene, with a background model whose main featuresconstitute spatially larger scales as well as lower frequencies. Thetask of matching foreground objects with their correct disparities thenbecomes simpler, given the relative consistency of background features.

Utilizing Micro-Electronic Machines (MEMS) for Adaptive Focus/Defocusand Aperture Size Modification

An alternative approach to enabling background defocus, or blurring isthrough changing the background model via varying the focal length bymounting the lens on microelectronic machines (MEMs). Therefore, as isshown in FIG. 7, a lens 710 is mounted to a MEM 720 allowing for varyinga distance between the lens 710 and a sensor 730. Once lens 710 ismounted, MEM 720 can modify the focal length based on optimal depthengine feedback. This can be performed iteratively with the depthengine. So, if the foreground segmentation quality is poor, a shallowerdepth of field may be accomplished by employing MEM 720 that allows lens710 to expand away from, or towards sensor 730, varying the focal lengthin real-time. Another advantage of utilizing MEMs lies in the ability todefine a narrow depth of field by varying both the aperture as well asthe focal length.

As a result, another approach can be suggested in which an artificialintelligence system, such as the one that has been described in theabove-mentioned '038, '055 and '070 applications, can be used toevaluate the quality of segmentation. The AI can then interactively varythe image by enhancing segmentation through a narrower, or shallowerdepth of field, in whichever configuration that the applicationrequires.

Therefore, in accordance with various embodiments of the presentinvention, a series of steps are provided for enhancing stabilitycriteria of computational stereo. Inventive segment definitions arepresented, as well as their transition criteria from unstable to stable,and between the various inventive additional segment definitions. Theconcept of computing depth on one or more residual components of animage sequence is also presented. Orthogonal decomposition of an imagesequence in the color space may enhance disparity decomposition byreducing the overall population of candidate pixels that can match for agiven disparity. A final depth map may be comprised of composites of allthe different depth maps that are produced in these orthogonalprojections. Additionally, depth of field of a scene may be manipulatedto highlight differences between the foreground and background andimprove depth computation through segment matching and backgroundmanipulation/modeling. A new, dynamic approach to varying the depth offield and the subsequent depth compute via MEMs is also presented.

API/SDK

In accordance with a further embodiment of the invention, an API ispresented that preferably takes advantage of the information providedfrom the depth computation, such that critical points, gesture events,as well as overall depth information is provided as part of the API.Additionally, an SDK is preferably presented such that softwaredevelopers can take advantage of these various features.

What is claimed:
 1. A method for segmenting an image, comprising thesteps of: determining one or more unstable segments of one or moresegments defined from a plurality of images changing by more than apredetermined threshold between two of the plurality of images;determining one or more transitioning segments transitioning from anunstable to a stable segment between two of the plurality of images;determining depth only for one or more of the one or more unstablesegments that have changed by more than the predetermined threshold andfor one or more of the one or more transitioning segments; and updatinga preexisting depth map with only the depth for the one or more unstablesegments and the one or more transitioning segments.
 2. The method ofclaim 1 wherein the transitioning segments comprise one or moremesostable segments.
 3. The method of claim 1, wherein the segments aredefined from a stereo pair of images.
 4. The method of claim 1, whereinthe determined depth for the one or more unstable and transitioningsegments replaces a prior corresponding depth value in the preexistingdepth map.
 5. The method of claim 1, further comprising the step ofgenerating a new depth map associated with the one of the plurality ofimages in accordance with the determined depth for the one or moreunstable segments, the one or more transitioning segments, and thepreexisting depth map.
 6. The method of claim 5, further comprising thestep of repeating the process for generating the new depth map of a nextsequential frame from the plurality of acquired images, such that onlythe changes in depth are computed.
 7. The method of claim 6, wherein adepth map associated with a next frame is generated by starting with thedepth map associated with the one of the plurality of images, andrecalculating depth associated with one or more unstable segments orportions of such segments that have changed by more than thepredetermined threshold and one or more transitioning segments.
 8. Themethod of claim 1, wherein the predetermined threshold may be learned.9. The method of claim 1, further comprising the step of combining oneor more transitioning segments with a stable segment to produce adifferent stable segment.
 10. A computer program stored to anon-volatile computer medium, the computer program causing amulti-purpose computer to segment an image by performing the steps of:defining one or more segments from one of a plurality of images;determining one or more unstable segments of the one or more segmentsdefined from a plurality of images changing by more than a predeterminedthreshold from a prior between two of the plurality of images;determining one or more transitioning segments transitioning from anunstable to a stable segment between two of the plurality of images;determining depth only for one or more of the one or more unstablesegments that have changed by more than the predetermined threshold andfor one or more of the one or more transitioning segments; and updatinga preexisting depth map with only the combining the determined depth forthe one or more unstable segments and the one or more transitioningsegments with a preexisting depth map.
 11. The computer program of claim10, wherein the transitioning segments comprise one or more mesostablesegments.
 12. The computer program of claim 10, wherein the segments areclustered from a stereo pair of images.
 13. The computer program ofclaim 10, wherein the determined depth for the one or more unstable andtransitioning segments replaces a prior corresponding depth value in thepreexisting depth map.
 14. The computer program of claim 13, furthercomprising the step of generating a new depth map associated with theone of the plurality of images in accordance with the determined depthfor the one or more unstable segments, the one or more transitioningsegments, and the preexisting depth map.
 15. The computer program ofclaim 14, further comprising the step of repeating the process forgenerating a depth map of a next sequential frame from the plurality ofimages.
 16. The computer program of claim 15, wherein a depth mapassociated with a next frame is generated by starting with the depth mapassociated with the one of the plurality of images, and recalculatingdepth associated with one or more segments that have changed more thanthe predetermined threshold and one or more transitioning segments. 17.The computer program of claim 16, wherein the predetermined threshold islearned.
 18. The computer program of claim 10, wherein one or more ofthe steps performed by the multi-purpose computer are performed by agraphical processing unit.
 19. The computer program of claim 10, furthercomprising the step of assigning cluster numbers to the one or moresegments according to their classification as unstable, transitioning orstable.
 20. The computer program of claim 19, further comprising thestep of changing the cluster number of a segment when the segmenttransitions from one classification to another.